Adjustment of sensor geometry model parameters using digital imagery co-registration process to reduce errors in digital imagery geolocation data

ABSTRACT

A digital image processing system reduces errors in the parameters of a sensor geometry model, through which points in a captured digital image are geolocated to the surface of the earth by means of a `real time` co-registration mechanism that refines the geometry model associated with the working image in a matter of seconds. Using a co-registration mechanism such as that described in the U.S. Pat. No. 5,550,937, the system co-registers the reduced accuracy working digital image with a reference image, geographical spatial locations of respective pixels of which have been previously determined with a high degree of accuracy. The imagery co-registration operator adjusts the respective geometry models associated with its input images, in accordance with differences in cross-correlations of the respectively different spatial resolution versions of the two images, so as to bring the respective images into effective co-registration on image registration surface. Mutual registration on the image registration surface of the working and reference images reduces the parameter errors in the original working image&#39;s sensor geometry model to the same error resolution of the reference image&#39;s geometry model.

FIELD OF THE INVENTION

The present invention relates to digital imagery processing systems inwhich digital images of areas of the surface of the earth are capturedby an image capture device, such as an electro-optical airborne cameraor radar system. The invention is particularly directed to a techniquefor reducing the degree of error and thereby improving the accuracy invalues of geolocated spatial coordinates of respective pixels of adigital image. It does this by adjusting sensor geometry modelparameters associated with the image capture device, employing a digitalimagery co-registration process that co-registers the digital image witha reference digital image, geographical spatial locations for respectivepixels of which have a degree of error significantly reduced withrespect to that of the captured digital image.

BACKGROUND OF THE INVENTION

A number of image capture systems, such as airborne or spaceborne cameraor radar systems, diagrammatically illustrated at 10 and 11,respectively in FIG. 1, are employed to capture images of areas 12 ofthe surface of the earth. In a number of applications, these images areused to locate one or more features of interest, in preparation forfurther activity, such as, but not limited to tactical theatre-basedinterdiction of one or more targets whose geographical locations mustnot only be determined with high accuracy, but may vary over arelatively brief time interval (e.g., on the order of only several ortens of hours), making time of the essence.

Because the image capture platform is typically mounted on areconnaissance aircraft 14 or the like, the parameters of an associatedsensor geometry model 15, through which a captured digital image 16 maybe related or transformed to the surface of (a digital elevation model(DEM) of) the earth containing the viewed area of interest, are not onlyaffected by the orientation of the image capture device, but by thesubstantial dynamics (including avionics errors) of the aircraft itself.If uncompensated, these offsets will introduce errors in geographicalcoordinates of respective points (pixels) in the digital image that areobtained by mapping or `geolocating` respective pixels (some of whichare shown at 17) of the digital image 16 to actual coordinates 21 (e.g.,latitude-Φ, longitude-γ and elevation-h) on the surface of the earth.

To solve this problem, it has been customary practice to have a skilledoperator at an image processing workstation 24 examine the display 25 ofthe `working` or input digital image 16 to locate what are known as`ground control points` 27. Such ground control points are those pointswhose actual geographical coordinates are known with a relatively highdegree of accuracy (e.g., to within one to five meters, or less), suchas may be obtained from a survey of the area of interest or from anarchival `reference` image 29 of the geographical area of interest. Byclicking on a display cursor 31 that has been manually positioned(mouse-manipulated) over a what is considered to be a respective groundcontrol point in the working image, the operator supplies to an offsetor error correction program within the workstation the apparent locationof the pixel, which is then compared by the correction program with theactual coordinates of the known ground control point in the referenceimage 29. By repeating this operation for numerous ground controlpoints, the operator sequentially supplies the image workstation'scorrection program with a relatively large number of data points, thatthe program uses to update or refine the parameters of the sensorgeometry model associated with the working image, and thereby reduceswhat is originally a relatively large geolocation offset in pixels ofthe working image to one that is closer to the error resolution of thereference image.

A fundamental problem with this operator-controlled error reductionscheme is the fact that it is extremely labor intensive (and therebysubject to an additional source of error--the operator), and timeconsuming, often taking hours to complete. If the image is onecontaining features whose locations are not necessarily static and mustbe acted upon within a relatively short period of time of theiridentification, the conventional operator-controlled approach may havelittle or no practical value to the ultimate user of the working image.Moreover, the conventional approach requires a reference image thatcontains a sufficient number of valid ground control points whoseaccuracy has been predetermined, such as a `survey` map. If such groundcontrol points have not been previously accurately located in such areference image, it may not be possible for the operator to obtain anymeaningful reduction in errors in the parameters used by the sensorgeometry model for the working image.

SUMMARY OF THE INVENTION

In accordance with the invention, the deficiencies of theabove-described labor and time-intensive conventionaloperator-controlled scheme for reducing errors in the parameters of asensor geometry model, through which points in a captured digital imageare geolocated or transformed to actual points on the surface of theearth, are remedied by what is effectively a `real time` co-registrationmechanism (that is able to refine the geometry model associated with theworking image in a matter of seconds).

As will be described, the inventive mechanism uses a digital imageryco-registration process that co-registers the reduced geolocationaccuracy `working` digital image with a more accurate reference digitalimage. The co-registration mechanism may be of the type described in theU.S. Pat. No. 5,550,937 (hereinafter referred to as the '937 patent), toD. Bell et al, entitled: "Mechanism for Registering Digital ImagesObtained from Multiple Sensors Having Diverse Image CollectionGeometries," assigned to the assignee of the present application and thedisclosure of which is herein incorporated.

In particular, the image processing scheme of the present inventioncouples a respective working image and its associated sensor geometrymodel to a digital imagery co-registration operator that is executedwithin an image processing workstation. Also coupled to the digitalimagery co-registration operator is a reference image that includes theterrestrial area of interest in the working image and an associatedsensor geometry model.

The reference image may be obtained from a variety of image captureplatforms, such as, but not limited to airborne or satellite-basedcamera, infrared sensor, radar units, etc., as described in the '937patent, and its selection is not necessarily based upon whether itcontains any ground control points (although it may be derived from suchan image). What is important is that the respective pixels of thereference image be geolocatable to points on the surface of the earth towithin the degree of accuracy required by the image exploiter. Themutual registration process described in the '937 operates on whateverpixels are contained in respective neighborhoods of pixels within theimage, rather than on particular pixels externally identified by anoperator.

The imagery co-registration operator is operative to iterativelycross-correlate content-emphasized neighborhoods of pixels ofrespectively different spatial resolution versions of the working imageand the reference image as projected onto an image registration surface.The image processor adjusts the respective geometry models associatedwith those images, in accordance with differences in cross-correlationsof the respectively different spatial resolution versions of the twoimages, so as to bring the working and reference images into effectiveco-registration on the image registration surface.

Since the geographical coordinates of the pixels of the reference imageas projected or transformed by the reference image's associated sensorgeometry model are within a relatively fine error resolution that isacceptable to the image exploiter, mutual registration on the imageregistration surface of the working and reference images will reduce theparameter errors in the original working image's sensor geometry modelto the same error resolution of the reference image's geometry model.Consequently, the geographical coordinates of any pixel in the mutuallyregistered working image will necessarily be as accurate as those in thereference image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 diagrammatically illustrates an image processing systemassociated with an airborne sensor platform which captures images of thesurface of the earth, containing one or more features whose geographicalcoordinates are to be accurately determined; and

FIGS. 2 and 3 diagrammatically illustrate an automated co-registrationbased image processing system in accordance with the present inventionfor processing digital images obtained by an image capture system whosesensor geometry model parameters do not allow sufficiently precisegeolocation of any pixel in the originally captured image with its truelocation on the earth.

DETAILED DESCRIPTION

Before describing in detail the image co-registration based sensorgeometry model error reduction scheme of the present invention, itshould be observed that the invention resides primarily in what iseffectively a prescribed digital imagery transformation operator, whichis preferably incorporated within the image processing software employedby a digital image processing system. Consequently, the configuration ofsuch a system and the manner in which it is interfaced with a digitalimage capture source have been illustrated in the drawings by readilyunderstandable block diagrams, which show only those specific detailsthat are pertinent to the present invention, so as not to obscure thedisclosure with details which will be readily apparent to those skilledin the art having the benefit of the description herein. Thus, the blockdiagram illustrations and the image processing diagrams of the Figuresto be described are primarily intended to show the major components ofthe system in a convenient functional grouping and processing sequence,whereby the present invention may be more readily understood.

FIGS. 2 and 3 diagrammatically illustrate an automatedco-registration-based, image processing system in accordance with thepresent invention for reducing the degree of error, and therebyimproving the accuracy in values of spatial coordinates of respectivepixels of respective digital images 100, that have been captured by animage capture system, the sensor geometry model for which containsinaccuracies that do not allow sufficiently precise geolocation of anypoint in the captured image with its true location on the earth. As anon-limiting example, such an image capture system may comprise a cameraor other type of image sensor platform, such as a synthetic apertureradar (SAR) 102 mounted on an aircraft 104 flying over a terrestrialarea of interest 106.

The terrestrial area is expected to contain one or more features (suchas non-static targets), the exact geographical location of which on thesurface of the earth (e.g., in terms of latitude, longitude andelevation) is desired by a downstream image exploiter 110. Typically,the task of the image exploiter 110 (who does not form part of theinvention) is to analyze the content of a respective image 100 forprescribed features by means of an image manipulation workstation 112.

As discussed above, because the features of interest may be non-static(mobile), determining their exact location as quickly as possible (asclose as possible to real time) is often essential to the success of thetask performed by the image exploiter. (As described above, because theconventional, manual process to compensate for the inaccuracies in thesensor geometry requires substantial operator participation and time tocomplete, if the features of interest are mobile, the eventually refinedor corrected version of the working image may be effectively `stale`, sothat it has little or no value to the image exploiter.)

To remedy this shortcoming, the present invention makes use of an imageco-registration mechanism to automatically adjust the sensor collectiongeometry model associated with the working image of interest,irrespective of the platform from which it was obtained, so that itsparameters have the same, relatively high degree of accuracy as those ofa geometry model associated with a co-registered reference image. As anon-limiting example, such an image co-registration mechanism maycomprise that described in the above-referenced '937 patent.

More specifically, as shown in the diagrammatic illustration of FIG. 3,rather than have an operator sequentially search for, locate and input arelatively large number of control points derived from a `ground controlpoint rich` archival reference image (which may not always be available,as described supra), the present invention couples a respective workingimage 121 as a first input image and its associated sensor geometrymodel 123 to a digital imagery co-registration operator 125 which may beinstalled within image processing workstation 112.

Also coupled to the digital imagery co-registration operator 125 is asecond (reference) image 131 (that includes the terrestrial area ofinterest in the first or working image 121) and its associated sensorgeometry model 133. The second image 131 may be accessed from a library130 of images that have been generated via a variety of image captureplatforms, such as, but not limited to airborne or satellite-basedcamera, infrared sensor, radar units, etc., as described in the '937patent, and its selection is not necessarily based upon whether itcontains any ground control points (although it may be derived from suchan image in the same manner as described above). Accessing the secondimage 131 from a library of reference images may be expedited by usinggeographical information, such as, but not limited to, the coordinatesof boundary corners of the working image 121, as (address) controlpointers to pages of images of the earth that contain at least the samearea as that bounded by the geographical coordinates of the workingimage.

What is key is that the second or reference image 131 be an image whoserespective pixels can be geolocated to actual points on the surface ofthe earth to within the degree of accuracy (e.g., within several secondsof a degree of latitude/longitude, within one to five meters ofelevation) required by the image exploiter. (Moreover, as a non-limitingexample, the reference image 131 may itself have been obtained byprocessing a plurality of images, derived from diverse image capturesensor platforms, in accordance with the mutual registration processdescribed in the '937 patent.)

It should be noted that, unlike a conventional archival reference image,which must contain a significant number of operator-discernible groundcontrol points, the reference image 122 need not be such an image, sincethe mutual registration process described in the '937 operates onwhatever pixels are contained in respective neighborhoods of pixelswithin the image, rather than on particular types of pixels that areexternally identified by an operator. Namely, as described in detail inthe '937 patent, the imagery co-registration operator 125 is operativeto iteratively cross-correlate content-emphasized neighborhoods ofpixels of respectively different spatial resolution versions of theworking image 121 and the reference image 131 as projected onto an imageregistration surface 140, and adjusts the respective geometry models 123and 133 associated with those images, in accordance with differences incross-correlations of the respectively different spatial resolutionversions of the two images, so as to bring the respective images 121 and131 into effective co-registration on image registration surface 140.

Because the spatial coordinates (in terms of latitude, longitude andelevation on the surface of the earth) of the respective pixels of thereference image 131, as projected or transformed by the referenceimage's associated sensor geometry model 133, are within a relativelyfine error resolution that is acceptable to the image exploiter, mutualregistration on the image registration surface 140 of the working andreference images 121 and 131 will result in a tuning or reduction inerrors in the original, `relatively sloppy` parameters of the sensorgeometry model 123 to the same error resolution of the reference image'sgeometry model 133. As a consequence, the spatial coordinates of anypixel in the working image 121 will necessarily be as accurate as thosein the reference image 131.

As will be appreciated from the foregoing description, the imageryco-registration operator 125 is operative to refine the geometry model123 associated with the working image 121 in a matter of seconds.Consequently, with the ability of present day satellite and airborneimaging and telemetry systems to rapidly capture and download digitalimages to an image processing workstation that incorporates the presentinvention, an image exploiter will have a `real time` image that allowsfollow-on tasks associated with features of interest to be completedwith a relatively high probability of success. This is a markedimprovement over the above-described conventional, manual process, whichrequires substantial operator participation and time to complete, andthereby may prevent the eventually corrected version of the workingimage from having any practical use to the image exploiter.

While we have shown and described an embodiment in accordance with thepresent invention, it is to be understood that the same is not limitedthereto but is susceptible to numerous changes and modifications as areknown to a person skilled in the art, and we therefore do not wish to belimited to the details shown and described herein, but intend to coverall such changes and modifications as are obvious to one of ordinaryskill in the art.

What is claimed:
 1. For use with a digital imagery processing system, inwhich a first digital image of an object is derived from a first imagecapture system, and wherein the position of a respective pixel of saidfirst digital image, in terms of a spatial coordinate system for saidobject, is definable to a first spatial accuracy, a method of improvingthe spatial accuracy of the location of the position of said respectivepixel of said first digital image, said method comprising the stepsof:(a) providing a first geometry model associated with said first imagecapture system that projects said first digital image onto aregistration surface; (b) providing a second digital image of saidobject that has been derived from a second image capture system,locations of respective pixels of which, in terms of said spatialcoordinate system, are known to a second spatial accuracy, said secondimage capture system having a second geometry model that projects saidsecond digital image onto said registration surface; and (c) modifyingparameters of said first geometry model by co-registering said firstdigital image with said second digital image on said registrationsurface in accordance with said first and second respective geometrymodels therefor, thereby causing locations of respective pixels of saidfirst image, in terms of said spatial coordinate system, to be known tosaid second spatial accuracy.
 2. A method according to claim 1, whereinsaid first and second digital images comprise first and secondrespective images of an area of the surface of the earth as captured bysaid first and second image capture systems, and wherein the positionsof respective pixels of said first and second digital images are definedin terms of latitude, longitude and elevation.
 3. A method according toclaim 1, wherein step (c) comprises iteratively cross-correlatingcontent-emphasized neighborhoods of pixels of respectively differentspatial resolution versions of said first and second digital images asprojected onto said registration surface, and adjusting said first andsecond respective geometry models in accordance with differences incross-correlations of said respectively different spatial resolutionversions of said first and second digital images as projected onto saidregistration surface, so as to bring said first and second digitalimages into effective co-registration on said registration surface.
 4. Amethod according to claim 1, wherein said first and second image capturesystems comprise diverse image capture systems.
 5. A method according toclaim 1, wherein step (b) comprises selecting said second digital imageof said object, from a library of digital images containing said object,based upon contents of said first digital image of said object.
 6. Amethod according to claim 5, wherein step (b) comprises selecting saidsecond digital image of said object, from a library of digital imagescontaining said object, based upon spatial coordinate information ofsaid first digital image of said object.
 7. For use with a digitalimagery processing system, in which a first digital image of an area ofthe surface of the earth is captured by a first image capture system,and wherein spatial coordinates on the surface of the earth of arespective pixel of said first digital image are definable to a firstdegree of error, a method of reducing said first degree of error andthereby improving the accuracy of values of spatial coordinates of saidrespective pixel of said first digital image, said first image capturesystem a first geometry model associated therewith that projects saidfirst digital image onto a registration surface, said method comprisingthe steps of:(a) providing a second digital image of said area of thesurface of the earth that has been derived from a second image capturesystem, locations of respective pixels of said second digital imagehaving spatial coordinates on the surface of the earth to a seconddegree of error, less than said first degree of error, said second imagecapture system having a second geometry model that projects said seconddigital image onto said registration surface; and (b) modifyingparameters of said first geometry model by co-registering said firstdigital image with said second digital image on said registrationsurface in accordance with said first and second respective geometrymodels therefor, thereby reducing said first degree of error of spatialcoordinates of respective pixels of said first image to said seconddegree of error.
 8. A method according to claim 7, wherein step (b)comprises iteratively cross-correlating content-emphasized neighborhoodsof pixels of respectively different spatial resolution versions of saidfirst and second digital images as projected onto said registrationsurface, and adjusting said first and second respective geometry modelsin accordance with differences in cross-correlations of saidrespectively different spatial resolution versions of said first andsecond digital images as projected onto said registration surface, so asto bring said first and second digital images into effectiveco-registration on said registration surface.
 9. A method according toclaim 7, wherein spatial coordinates of pixels of said first and seconddigital images of said area of the earth are defined in terms oflatitude, longitude and elevation.
 10. A method according to claim 7,wherein said first and second image capture systems comprise diverseimage capture systems.
 11. A method according to claim 7, wherein step(a) comprises selecting said second digital image, from a library ofdigital images containing said area of the surface of the earth, basedupon contents of said first digital image.
 12. A method according toclaim 11, wherein step (a) comprises selecting said second digitalimage, from a library of digital images containing said area of thesurface of the earth, based upon spatial coordinate information of saidfirst digital image.
 13. For use with a digital imagery processingsystem, in which a first digital image of an area of the surface of theearth is captured by a first image capture system having a first,associated geometry model through which geographical coordinates on thesurface of the earth of a respective pixel of said first digital imageare definable to a first degree of error, a digital image processingarrangement for reducing said first degree of error and therebyimproving the accuracy of values of geographical coordinates of saidrespective pixel of said first digital image comprising:a second digitalimage of said area of the surface of the earth that has been derivedfrom a second image capture system, locations of respective pixels ofsaid second digital image having spatial coordinates on the surface ofthe earth to a second degree of error, less than said first degree oferror, and a second geometry model through which said respective pixelsof said second digital image are projectable onto a registrationsurface; and a digital image processor which is coupled to receive firstdata representative of said first digital image and said first,associated geometry model, and second data representative of said seconddigital image and said second geometry model, and being operative tomodify co-register said first digital image with said second digitalimage on said registration surface in accordance with said first andsecond respective geometry models therefor, thereby reducing said firstdegree of error of spatial coordinates of respective pixels of saidfirst image to said second degree of error.
 14. A digital imageprocessing arrangement according to claim 13, wherein said digital imageprocessor is operative to iteratively cross-correlate content-emphasizedneighborhoods of pixels of respectively different spatial resolutionversions of said first and second digital images as projected onto saidregistration surface, and to adjust said first and second respectivegeometry models in accordance with differences in cross-correlations ofsaid respectively different spatial resolution versions of said firstand second digital images as projected onto said registration surface,so as to bring said first and second digital images into effectiveco-registration on said registration surface.
 15. A digital imageprocessing arrangement according to claim 13, wherein spatialcoordinates of pixels of said first and second digital images of saidarea of the earth are defined in terms of latitude, longitude andelevation.
 16. A digital image processing arrangement according to claim13, wherein said first and second image capture systems comprise diverseimage capture systems.
 17. A digital image processing system accordingto claim 13, wherein said second digital image is one that has beenselected from a library of digital images containing said area of thesurface of the earth, based upon contents of said first digital image.18. A digital image processing system according to claim 17, whereinsaid second digital image is one that has been selected from a libraryof digital images containing said area of the surface of the earth,based upon spatial coordinate information of said first digital image.